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I.  INTRODUCTION 


Anthropometric  data  have  been  used  by  the  Air  Force  for  many  years 
to  help  in  the  design  of  clothing,  equipment,  cockpits,  etc.  The 
use  of  anthropometric  statistics  enables  designers  to  provide  a 
better- fitting  product  and  ensure  that  the  people  who  will  be  using 
the  equipment  are  physically  able  to  do  so. 

Traditionally,  the  tools  of  anthropometry  have  been  physical 
measuring  devices  such  as  calipers  and  tape  measures.  Recentlv, 
the  Homan  Engineering  Division  at  Wright-Patterson  Air  Force  Base 
has  been  exploring  the  use  of  3 -dimensional  scanning  technology  to 
produce  a  digital  representation  of  the  surface  of  the  human  head. 

A  vertical  stripe  of  laser  light  is  projected  ontc  a  stationary 
subject  as  the  scanner  rotates  360  degrees  around  him  or  her. 
taking  video  recordings  which  are  converted  into  digital  format. 
The  digitized  image  is  stored  as  a  set  of  points  representing  the 
distance  from  the  center  of  rotation  to  the  surface  of  the  subject 
at  256  points  along  each  longitude.  This  technique  has  been  used 
to  collect  data  on  both  men  and  women  to  be  used  in  the  design  of 
such  equipment  as  night  vision  goggles  for  Air  Force  pilots. 

-  Although  the  data  collected  in  this  way  are  much  more  complete  than 
was  possible  using  traditional  methods,  techniques  have  not  yet 
been  developed  to  easily  analyze  and  use  this  type  of  data..  Due  to 
the  large  size  and  noisiness  of  the  data  set,  and  the  difficulty  of 


recognizing  facial  features  which  vary  greatly  in  size  and  shape 
among  individuals,  the  use  of  artificial  intelligence  methodologies 
seems  appropriate  to  the  problem. 

My  academic  background  incorporates  two  specialties  that  are 
directly  relevant  to  the  problem  described  above.  My  Ph.D.  is  in 
physical  anthropology ,  which  is  the  academic  discipline  that 
includes  the  study  of  anthropometry.  More  recently,  I  received  a 
M.S.  degree  in  computer  science,  with  a  specialty  in  'artificial 
intelligence.  1  have  been  involved  in  AI  research  for  the  past 
several  years.  These  factors  led  to  my  assignment  to  the 
AAMRL/Human  Engineering  Division  for  the  summer  research  program. 

II.  OBJECTIVES  OF  THE  RESEARCH  EFFORT 


One  major  problem  already  encountered  in  the  use  of  3-D  data  is  the 
identification  of  facial  landmarks  on  the  image.  The  landmarks  are 
used  to  define  boundaries  between  different  regions  of  the  head, 
and  to  serve  as  ei  dpoints  for  distance  measurements .  There  are 
approximately  40  landmarks  that  must  be  recorded  for  each  image  and 
stored  as  a  set  of  longitude/latitude  coordinates. 

Currently,  about  half  the  landmarks  are  being  marked  on  the 
subjects  prior  to  scanning  using  small  felt  patches  which  do  not 
reflect  the  laser  light.  The  remaining  landmarks  are  those  which 
can  be  visually  identified  on  the  image,  such  as  the  tip  of  the 
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nose,  or  those  which  cannot  be  marked  with  patches,  such  as  the 
pupil  or  the  stomion  (center  of  the  mouth  along  the  meeting  line 
between  the  two  lips).  Someone  must  then  record  the  longitude  and 
latitude  of  each  landmark  in  a  separate  file  associated  with  the 
image  file.  This  is  done  by  using  the  cursor  to  locate  each 
landmark  on  a  graphic  representation  of  the  head  and  instructing 
the  program  to  record  the  latitude  and  longitude  of  the  point 
identified  by  the  cursor. 

A  means  of  having  the  landmarks  automatically  identified  by  an  Al 
program  would  save  a  great  deal  of  time  for  both  the  technicians 
who  currently  identify  the  landmarks  manually,  and  for  the  subjects 
being  scanned.  The  goal  would  be  *-0  eliminate  the  need  for  marking 
subjects  with  patches,  or  to  reduce  those  to  as  few  as  possible, 
and  to  have  a  computer  program  identify  and  record  the  coordinates 
of  each  landmark.  In  addition  to  saving  time,  computerizing  this 
process  would  ensure  consistent  landmark  identification  across 
subjects,  which  is  especially  difficult  when  more  than  one  person 
does  the  marking. 

My  goals  for  the  1989  Summer  Faculty  Research  Program  were  (1)  to 
survey  the  scientific  literature  for  techniques  that  might  be 
applicable  to  the  problem  described  above,  and  (2)  to  test  the  most 
promising  approaches,  and  (3)  if  any  approach  appeared  to  be 
successful,  to  propose  a  general  design  for  a  system  that  would  do 
the  described  tasks. 
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III. 


A  survey  of  the  computing  literature  revealed  several  possible 
approaches  to  the  problem.  There  is  a  large  literature  on  computer 
vision,  but  most  work  in  that  field  is  inappropriate  to  this  domain 
for  one  of  the  following  reasons:  (1)  It  deals  with  issues  of 
creating  3 -dimensional  models  from  2-dimensional  images.  In  this 
research,  the  3-D  image  is  the  initial  form  of  the  data  and 
therefore  those  issues  are  not  relevant.  (2)  Much  of  the  computer 
vision  work  takes  a  highly  mathematical  approach  to  analyzing 
curves,  surfaces ,  etc.  For  the  landmark  identification  problem,  a 
knowledge -based  approach  using  contextual  information  about  the 
face  and  relevant  features  appears  to  be  more  useful.  This 
approach  is  found  in  the  work  by  Kjeldsen,  Califano  and  Bolle 
(1989)  on  object  recognition  in  a  3-D  framework. 

The  most  useful  areas  in  the  scientific  literature  are  in  the 
categories  of  pattern  recognition  and  artificial  intelligence.  The 
neural  network  approach,  as  described  by  Pao  (1989),  has  potential 
for  contributing  to  the  landmark  identification  problem,  but  there 
are  some  problems  to  be  overcome.  First,  the  large  size  of  the 
data  set  for  even  one  image  (a  312  X  256  array,  or  131,072  points) 
requires  an  extremely  large  network  of  neurons  for  processing,  and 
therefore  would  require  large  measures  of  both  time  and  memory. 
Second,  the  problem  at  hand  would  seem  to  require  two  levels  of 
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network  processing:  a  first  pass  might  identify  facial  features, 
and  a  second  pass  could  be  used  to  locate  landmarks  in  relation  to 
those  features.  Third,  it  difficult  to  tell  whether  a  neural 
network  approach  is  capable  of  solving  this  problem  without 
building  the  entire  system. 

The  approach  that  appears  to  be  the  most  promising  is  a  knowledge  - 
based  system  using  a  blackboard  architecture.  This  type  of  svstem 
has  been  applied  to  symbol -process ing  problems  that  are  in  some 
wavs  analogous  to  landmark  identification.  One  major  advantage  of 
the  blackboard  approach  is  the  ease  with  which  various  types  of 
data  may  be  integrated  into  the  system.  and  the  use  of  neural 
network  technology  within  a  blackboard  system  is  a  viable 
alternative . 

Blackboard  systems  are  characterized  by  more  sophisticated  control 
structures  and  data  representation  facilities  than  are  found  in 
more  traditional  expert  systems.  They  usually  consider  a  problem 
from  multiple  levels  of  abstraction,  with  modules  at  one  level 
evaluating  hypotheses  generated  by  modules  at  other  levels  and 
generating  their  own  hypotheses  in  turn.  Blackboard  systems 
usually  use  'opportunistic'  reasoning,  which  gives  them  the  ability 
to  vary  the  problem  solving  algorithm  according  to  the  way  in  which 
the  solution  is  developing.  They  are  highly  flexible  systems  and 
have  been  used  to  solve  such  problems  as  speech  recognition  (Erman, 
et.al.  1980),  sonar  signal  analysis,  (Nii,  et.al.  1982),  and 


analysis  of  aerial  photographs  (Nagao,  et.al.  1979).  A  good 
overview  of  the  blackboard  architecture  and  applications  may  be 
found  in  Engelmore  and  Morgan  (1988). 


IV. 


A  blackboard  system  will  work  only  if  there  are  modules  capable  of 
generating  reasonable  hypotheses  about  the  data.  The  feasibility 
of  the  blackboard  approach  by  was  tested  this  summer  bv  the 
development  of  several  such  modules. 

The  first  module  looks  at  poirts  on  individual  longitudes  in  an 
attempt  to  locate  facial  features.  These  points  are  the  radii 
(distance  from  the  center  of  rotation)  that  have  been  measured  at 
each  latitude.  Traversing  a  single  longitude,  one  can  discern 

lucal  maxima  and  minima  in  the  radii  which  represent  protrusions 
and  indendations  respectively.  For  this  purpose.  a  local  minimum 
is  a  point  whose  neighbors  both  have  larger  values:  a  locai  maximum 
Is  a  point  whose  neighbors  both  have  smaller  values. 

Facial  features  that  show  up  along  a  longitude  as  maxima 

include  the  chin,  nose,  upper  and  lower  lips,  etc.  Features  that 
show  up  as  minima  include  the  mouth  (which  appears  as  an 
indentation  between  the  two  lips),  the  eves,  and  the  center  of  the 
ear.  In  each  of  these  cases,  the  local  maximum  or  minimum  occurs 
across  a  number  of  longitudes  at  approximately  the  same  latitude. 
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However,  due  to  errors  introduced  by  the  imaging  and  digitization 
r  jcess,  the  maximum  or  minimum  typically  does  not  show  up  on  all 
stripes.  An  additional  complication  is  that  most  features  are  not 
horizontal  lines  but  have  a  curved  or  angled  shape  and  show  up  at 
gradually  changing  latitudes.  For  example,  following  the  line  of 
the  chin  from  the  center  to  either  side  will  take  you  to  from  lower 
to  higher  latitudes. 

These  problems  were  successfully  solved  in  Module  A  which  is 
capable  of  recognizing  a  feature  as  a  pattern  of  local  maxima  or 
minima  that  can  be  followed  across  a  set  of  adjacent  longitudes. 
The  features  discovered  by  this  module  have  no  semantic  information 
associated  with  them.  That  is.  Module  A  does  not  recognize  that 
Feature  X  is  a  nose  or  a  chin.  Many  of  the  features  found  in  this 
way  do  not,  in  fact,  represent  facial  features  that  are  being 
sought.  The  ear,  for  example,  has  a  large  number  of  maxima  and 

minima  that  this  approach  identifies  as  features,  while  they  are 
really  part  of  one  larger  feature.  The  fact  that  some  meaningful 
features  are  recognizable,  however,  is  a  major  step  toward  landmark 
identification.  One  way  to  locate  the  promenton  (most  protrusive 
point  on  the  chin)  is  to  identify  the  chin  as  a  gross  feature  and 
take  the  center  point.  This  technique  applies  to  a  number  of  other 
landmarks  as  well.  The  difficult  part  of  the  process  is 

identifying  which  min/max  feature  is  actually  the  chin. 

Module  B  was  developed  to  look  at  possible  features  identified  by 
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Module  A  and  generate  hypotheses  regarding  which  facial  features 
they  might  represent,  and  to  also  generate  hypotheses  about  the 
positions  of  landmarks  on  those  features.  In  a  full  blackboard 
system,  Module  B  would  have  access  to  information  about  the 
positions  of  human  facial  features  relative  to  one  another.  In 
this  prototype,  that  information  is  supplied  by  the  user.  When 
Module  B  generates  2  or  3  hypotheses  about  a  particular  landmark, 
the  user  chooses  the  correct  hypothesis  (if  a  correct  hypothesis  is 
generated) ,  and  that  information  is  used  by  Module  B  in  the 
generation  of  hypotheses  for  other  landmarks.  The  results  obtained 
using  this  prototype  are  given  in  Table  1. 

Some  landmarks  are  not  associated  with  features  that  show  up  as 
local  maxima  or  minima.  For  example,  the  sellion  is  the  point  on 
the  central  profile  between  the  nose  and  the  forehead  where  the 
radius  is  at  a  minimum.  In  many  people,  however.  this  is  not  a 
local  minimum.  but  a  flat-bottomed  va^ev.  It  frequently  does  not 
show  up  as  a  minimum  using  the  algorithm  described  above. 

Another  way  to  approach  the  data  is  to  look  at  a  the  points  on  a 
longitude  as  a  set  of  slopes  computed  as  the  differences  between 
adjacent  radii.  One  can  then  identify  points  at  which  the  slope 
changes,  which  may  help  to  identify  landmarks.  The  sellion  is  an 
example  of  a  landmark  that  is  more  easily  identified  in  this  way. 
A  pattern  can  be  seen  in  which  sellion  is  clearly  discernable  as 
the  slope  changes  between  the  nose  and  the  forehead.  Changes  in 
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236/115 

s91 

298/152 

298/143 

295/105 

294/95 

310/123 

296/127 

284/125 

298/152 

296/144 

294/105 

294/95 

318/120 

296/127 

274/123 

s92 

288/141 

288/134 

288/92 

288/83 

313/113 

287/117 

266/115 

I 

288/142 

287/118 

288/79 

288/69 

323/111 

287/118 

251/113 

1  s93 

301/134 

302/125 

299/88 

299/79 

319/106 

302/108 

287/103 

1 

301/134 

300/125 

300/89 

300/79 

322/105 

300/109 

279/106 

I  s95 

309/140 

311/137 

307/95 

307/86 

324/111 

308/115 

295/111 

310/141 

310/132 

309/95 

309/85 

329/108 

308/115 

287/111 

Table  1.  Recorded  (top  row)  and  computed  (bottom  row)  landmarks  for 
randomly  chosen  male  subjects. 


Pro- 

nasale 

Sub- 

nasale 

Pro- 

menton 

Menton 

Right 
Che.  lion 

S  tomion 

Left 

Chelion 

s99 

281/139 

281/139 

282/130 

282/128 

281/95 

275/92 

280/84 

275/82 

304/108 

320/104 

281/113 

284/113 

257/108 

248/106 

sl33 

277/102 

277/106 

277/98 

277/135 

278/61 

277/63 

277/53 

277/53 

295/82 

299/80 

277/85 

277/85 

263/82 

256/82 

sl60 

298/130 

298/130 

298/122 

298/123 

296/85 

293/87 

297/76 

293/77 

321/101 

324/99 

297/105 

298/105 

275/103 

273/102 

sl76 

306/126 

306/127 

306/119 

306/116 

306/84 

307/85 

306/74 

307/75 

322/99 

324/97 

306/104 

306/102 

292/100 

288/99 

s!99 

300/134 

299/133 

300/122 

299/126 

301/78 

303/83 

302/72 

303/73 

317/102 

318/104 

300/106 

299/108 

289/103 

231/104 

Table  1.  (concinued) 


slope  are  useful  in  identifying  a  number  of  other  landmarks  as 
well,  such  as  the  promenton,  subnasale  and  pronasale. 

Module  C  tests  this  approach  to  analyzing  the  data.  Given  the 
longitude  representing  a  central  profile,  it  is  able  to  identify 
the  sellion,  pronasale,  subnasale  and  promenton.  Since  it  is 
possible  to  generate  hypotheses  regarding  the  latter  three 
landmarks  by  both  modules  B  and  C,  a  module  operating  at  a  higher 
level  of  analysis  in  a  complete  blackboard  system  would  be  able  to 
check  for  consistency  and  therefore  evaluate  the  hypotheses  with 
more  confidence. 


V.  RECOMMENDATIONS 


The  prototyping  effort  was  judged  to  be  successful  in  demonstrating 
that  modules  can  be  developed  to  generate  reasonable  hypotheses 
regarding  the  positions  of  a  select  group  of  landmarks  on  a  sample 
of  20  subjects  in  one  pose.  Although  these  results  are  narrow  in 
scope,  they  are  sufficiently  positive  to  justify  further  work  on 
this  project. 

The  design  of  a  blackboard  system  should  be  undertaken  to  implement 
the  ideas  described  in  this  report.  In  addition  to  the  modules 
implemented  in  this  work,  other  techniques  that  can  be  integrated 
into  the  blackboard  system  include  the  following: 
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a.  Knowing  the  approximate  longitude  of  the  center  profile 
improves  the  performance  of  the  existing  modules .  A  module  should 
be  developed  to  identify  this  region  with  some  certainty  before 
attempting  to  find  any  landmarks.  This  module  may  be  implemented 

as  a  neural  network  system  which  has  been  trained  on  the  pattern  of 
slopes  on  central  profile  longitudes.  The  longitudes  of  a  new 
image  can  then  be  run  through  the  network,  and  the  one  most 
resembling  the  previously  seen  central  profiles  can  be  identified. 
It  will  be  necessary  to  test  this  technique  in  order  to  determine 
if  it  alone  is  sufficient  to  identify  the  central  profile.  or  if 
additional  analysis  is  necessary  to  confirm  its  results. 

b.  Analysis  of  the  relationships  among  the  landmarks  should  be 
undertaken  to  determine  if  there  are  landmarks  whose  positions  can 
be  identified  by  reference  to  other  landmarks.  For  example,  in  the 
sample  of  subjects  used  in  the  prototype  testing,  the  program  was 
able  to  locate  the  menton  to  within  3  latitudes  simply  by 
subtracting  10  from  the  latitude  of  the  promenton.  If  this 
relationship  holds  for  a  larger  sample,  it  will  greatly  simplify 
the  process  of  locating  the  menton.  Testing  should  be  done  to 
ascertain  if  similar  relationships  exist  among  other  sets  of 
landmarks  . 

c.  A  constraint  network  should  be  incorporated  into  the  system 
which  contains  the  positional  relationships  among  facial  features 
and  landmarks.  This  can  be  used  to  identify  landmark  locations  as 
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described  in  (b)  .  However,  it  will  also  be  valuable  in  the 
evaluation  of  hypotheses.  If,  for  example,  the  central  profile 
longitude  has  been  identified  with  high  certainty,  a  hypothesis 
which  places  the  ectocanthus  (outer  corner  of  the  eye)  along  that 
longitude  should  be  quickly  rejected  because  its  position  does  not 
satisfy  the  constraints  of  these  relationships. 

d.  A  method  for  handling  uncertain  knowledge  should  be 
incorporated  into  the  system.  This  should  probably  take  the  form 
of  certainty  or  confidence  factors  as  used  in  many  expert  systems. 
However,  other  methods  for  handling  uncertainty  may  be 
investigated. 

e.  For  those  landmarks  whose  positions  can  be  determined  only  by 
palpation  and  marked  with  a  felt  patch,  a  module  should  be 
implemented  to  recognize  the  voids  in  the  data  set  which  represents 


those  landmarks. 
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